1. Field of the Invention
In general, the present invention relates to structured light systems that utilize multi-pattern techniques, whereby multiple projected structured light patterns are used to reconstruct an image. More-particularly, the instant invention is directed to a technique and system that employs at least one camera and one projector used in concert for retrieving depth information about at least one surface of an object employing traditional, and new, structured light pattern projections. The unique method, system, and program code of the invention incorporate the projection of a composite image comprising a plurality of modulated structured light patterns, at an object. Recovery of initial pattern information from an image reflected from the object is done for each of the modulated structured light patterns, in a manner that preserves depth information within each recovered pattern. From the depth information, the surface can be reconstructed by way of producing a depth map/mapping thereof.
While many of the known multi-pattern techniques address problems of depth ambiguity, invariance to surface reflectance variations, and depth accuracy, when using the traditional technique of projecting multiple patterns to reconstruct an image in motion, it takes an inordinately long time to capture all the necessary information for reconstruction. The consequence of this is that these known multi-pattern techniques become quite sensitive to object movement during the projection capture process. To address this problem, prior attempts have been made by others to increase the projection/capture rate. Unfortunately, as the projection/capture rate is increased the capture duration decreases the amount of light captured. Synchronization becomes more critical and the SNR of the capture images decreases. More-particularly the applicants' invention is directed to utilizing the spatial dimension that is orthogonal (herein, simply “orthogonal dimension”) to the depth distortion (i.e., “phase dimension”) to modulate and combine multiple pattern projections into a single composite pattern. Applicants' hereby disclose a new technique that offers a flexible way to systematically combine multi-patterns of structured light obtained using any selected technique, into a single composite pattern that can be continuously projected—thus, minimizing delay between pattern projections—providing for real-time 3D video imaging. Preferably, the multi-patterns are obtained using techniques that benefit from current solutions to depth ambiguity, invariance to surface reflectance variations, and depth inaccuracy. Using a composite pattern projected according to the invention, applicants' have identified a way to identify position and perform face recognition.
The composite pattern technique of the invention enables a wide range of multi-image methods to be utilized on moving objects. By modulating multiple structured light images into separate “channels” along the orthogonal dimension, bandwidth is utilized-just as in communications systems—resulting in a resolution tradeoff along the orthogonal dimension. To further refine the applicants' unique technique, of particular focus is any limitation(s) imposed by current state of the lithographic technology used to make the composite pattern, the number of pixel units along the orthogonal dimension of current camera technology, and monochromatic aberrations traditionally encountered using available optics technology.
2. Discussion of Background Technology: Classic Structured Light Projection
Typically, light structures function by projecting a predefined source onto a surface and then mapping the corresponding distortion in the light structure to surface features. Structured light measurement techniques provide a useful means by which data about a 3D surface may be acquired without contacting the surface, and used for a wide variety of engineering and industrial applications.
Structured light is the projection of a light pattern (plane, grid, or more complex shape) at a known angle onto an object. ‘Light’ and ‘images’ thereof, as used herein, includes electromagnetic (EM) energy radiated throughout the EM spectrum, and more preferably, within the spirit and scope of the invention, while the full EM spectrum is available for carrying out the invention, the focus is on EM emission(s) which fall within an extended range from the ultraviolet category (wavelengths from ˜180 nm) through near-infrared (NIR) category (wavelengths from-2000-n). In the event EM radiation falling outside the range of ˜180 run through ˜2000 nm is used according to contemplated features of the invention, equipment capable of projection and capture of the radiation must be selected to accommodate that radiation. For example, if acoustic frequencies are employed, associated acoustic transmitter and receiving units must be selected for projection and capture of the composite ‘image’ information for processing to recover the structured ‘light’ patterns initially modulated to compose the composite that is projected/transmitted at the surface of an object.
One type of traditional light patterning often used in process control machine vision is generated by fanning out a light beam into a ‘sheet’ of light. When the sheet of light intersects with an object, a bright line of light can be seen on the surface of the object. By viewing this line of light with a camera oriented at an angle, the observed distortions in the line can be translated into height variations. Scanning the object with the light constructs 3D information about the shape of the object, often referred to as active triangulation. This is the basic principle behind depth perception for machines, or 3D machine vision. Since structured lighting can be used to determine the shape of an object in machine vision applications, as well as help recognize and locate an object in other environments. Structured lighting has proven useful in assembly lines implementing process or quality control, by offering an automatic means to check for alignment and component breakage or condition, for example. Stocker Yale, Inc. distributes an off-the-shelf LASIRIS™ laser projector useful for process control in manufacturing lines to carry out component inspection and alignment.
As pointed out in Daley and Hassebrook, “Improved Light Sectioning Resolution by Optimized Thresholding,” SPIE Proceedings, 2909, 151-160 (November 1996), traditional approaches to structured light illumination include light stripe projection onto a surface topology and then analyzing the lateral displacements of the reflected pattern to reconstruct the surface topology. While a single spatial frequency of a light stripe pattern may be used to illuminate a relatively flat surface, in the case of rough surfaces, the surface topology is preferably encoded with a sequence of light stripe patterns with successively higher spatial frequencies. Either way, maximum resolution is limited by the maximum spatial frequency used. As spatial frequency increases, the projection system's blurring function causes the light stripes to be coupled thereby decreasing the SNR of the reflected image.
Correctly relating distortion to surface features is fundamental to structured light techniques. Discussion of how this has traditionally been addressed can be found in the technical manuscript of Raymond C. Daley, entitled “Design, Implementation and Analysis of Structured Light Systems,” (1997)—hereafter “Daley (1997)”—submitted in fulfillment of a Masters of Science degree, and labeled ATTACHMENT B as incorporated by reference in applicants' pending provisional application for background technical discussion. One fundamental geometric mechanism for creating the distortion is depicted in FIG. 1 of Daley (1997): it is the triangulation which consists of a source incident on a surface at a known angle, displaced laterally relative to a fixed viewing perspective, by the surface. Lateral displacement Δx can be found by viewing or measuring the location of the reflected source, and surface height deviation Ah is determinable by trigonometry.
FIG. 6 of Daley (1997) illustrates an example of a simulated single-stripe surface scan where the surface has been modeled as a 3D Gaussian pulse and the stripes modeled as a light plane intersecting the surface at a fixed projection angle. Only one light plane is depicted in FIG. 6, Daley (1997), representing the stripe projected onto the surface, while multiple intersections between the light strip and the surface are shown to represent a scan across the surface. From Daley (1997) FIG. 6 one can see how the stripes map to the surface contour. For each stripe image received, only one slice or section of the surface can be reconstructed. In order to reconstruct the entire surface, it is necessary to move the surface or projector and receiver such that the stripe is projected and viewed at each point on the surface to be measured. This is quite a cumbersome process.
FIG. 7 of Daley (1997) shows how the stripe projections from the simulated scan in FIG. 6, might appear if viewed by the receiver positioned directly above the surface. This 2D image represents lateral offsets which correspond to displacement Δx from Daley (1997) FIG. 1, thereby allowing surface height to be determined at the scanned intervals (pre-registration/calibration required). Daley (1997) discusses several of the difficulties encountered in converting a 2D stripe image into 3D data. A technique called multi-stripe projection extends single-stripe system by scene illumination with multiple slits or stripes that may be spatially modulated. Daley (1997) beginning at page 21, and pages 73-79 discusses two major components to structured light systems, namely, the projection and imaging subsystems designed to transmit and capture reflected light while maintaining or enhancing information in the light structure.
FIGS. 5 and 6 of Hassebrook, Daley, and Chimitt, “Application of Communication Theory to High Speed Structured Light Illumination,” Edited by Harding and Svetkoff, SPIE Proceedings, 3204(15), 102-113 (October 1997), concerns a structured light technique advancement made by at least one of the applicants hereof. Hassebrook, et al. (October 1997) draws an analogy between the projection of a structured light pattern to encoding a surface with a spatial carrier “image” analogous to a carrier signal from communications theory. Structured light systems were treated as wide bandwidth parallel communications channels. Surface characteristics (including topology) act to modulate the carrier image. Known SLM (spatial light modulator) devices allow one to program the encoding of the surface with a sequence of encoded patterns, as depicted in FIG. 5 and FIG. 6 of Hassebrook, et al. (October 1997).
In an effort to improve structured light systems (to which communications analogies were applied) with an eye toward optimizing spatial frequency while maintaining a fixed range resolution of the light-stripes, a technique was presented to enhance lateral resolution by multiplexing the light structure to produce interlaced encoded images, see Daley and Hassebrook, “Channel capacity model of binary encoded structured light-stripe illumination,” Applied Optics, 37(17), 3689-3696, June (1998). FIG. 8 therefrom illustrates a first set of eight different stripe patterns used to encode the object; the stripe pattern having the highest spatial frequency shown at the upper left in FIG. 8 (spatial period of 8 pixels/cycle). A second set of stripe-encoded images was captured identical to that shown in FIG. 8 except that all the stripe patterns are offset ¼-wavelength of the highest stripe frequency (¼*8, or 2 pixels/cycle). As explained by Daley and Hassebrook, June (1998) the two sets of images are binarized. An encoded image, one for each set, is generated as a weighted sum of the eight binary images. To interlace the two encoded images, stripe edge locations were determined. By combining the low bit images from both sets, a four-level encoded image was obtained, as illustrated in FIG. 9 of Daley and Hassebrook, June (1998).
As mentioned, known structured-light illumination techniques used for automated inspection and measuring surface topologies, are cumbersome. Traditional 3D acquisition devices use a single scanning laser stripe scanned progressively over the surface of the target object, placing a burden on the object to remain static and a burden on data acquisition to capture all the stripe images. In an attempt to reduce computational burdens of scanning and processing each scan position of the laser stripe, certain methods have been devised to project and process structured-light patterns, such as multi-stripe and sinusoidal fringe patterns, that illuminate the entire target surface at the same time. Specifically, one known technique focused at addressing the ambiguity and the albedo problems is to encode the surface repeatedly with multiple light striped patterns with variable spatial frequencies. These known multi-stripe patterns suffer from drawbacks such as introducing ambiguities in the surface reconstruction around surface discontinuities, overly sensitive to surface reflectance variations (i.e., albedo), and/or they suffer from lower lateral resolution caused by the required spacing between stripes. These known systems are simply not suitable for real-time imaging.
Thus, and according to the invention, a very unique approach is outlined herein as supported by rigorous mathematical and engineering analyses performed by the applicants. Applicants have discovered a structured-light patterning that allows, with a single projected composite image, the measuring of surface topologies that addresses issues of ambiguities, higher accuracy, and less sensitivity to albedo variations.